"The highlighted tokens are primarily Chinese, Japanese, and some corrupted or missing characters, often marking key nouns, verbs, or morphemes that carry core semantic meaning in a sentence, such as names, actions, or important objects. These tokens frequently appear at the start of compound words or phrases, and are often associated with high informational content or serve as grammatical anchors in the text."
Score Type | Accuracy | Precision | Recall | F1 score | TPR | TNR | FPR | FNR |
---|---|---|---|---|---|---|---|---|
detection | 0.7 | 0.917 | 0.44 | 0.595 | 0.44 | 0.96 | 0.04 | 0.56 |
fuzz | 0.89 | 0.882 | 0.9 | 0.891 | 0.9 | 0.88 | 0.12 | 0.1 |